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ABSTRACT 

This paper identifies and describes techniques that 
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Basic Education (ABE) Special Projects, and a list of principles and 
diagnostic questions from ABE staff development self-study, "as well 
as an annotated bibliography of six documents on relevant 
administrative and evaluation questions that are readily available to 
statcr adult education programs. (KG) 
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Introduction 

The Adult Education Act (PL 100-297) takes effect on July 1, 1988. 
This legislation and regulations require that States evaluate their Sec. 
353 (formerly Sec. 310) program at least once every three years. A few 
States conduct extensive evaluations of their systons and projects. 
Most, however, develop a narrative report based on monitoring observa- 
tions and project reports. Others do no more than describe what was 
funded. 

The purpose of this paper is to identify and promote techniques that 
States may use to evaluate the three coitponents of their Sec. 353 
program: 

1. State administration, 

2. Special projects and staff training programs, 

3. Effects or inpacts of projects on local programs and 
instructors. 

It includes an overview of techniques for each of ttie three conponents 
and an Appendix containing evaluation processes and approaches as well as 
an annotated bibliography of more extensive documents that are readily 
available to State Adult Education Programs. 

Consultation, information,, and documents addressing Sec. 353 evaluation 
•are also available from: 

James Parker 

Division of Adult Education 
U.S. Dept. of Education 
Washington, DC 2e?.02-5515 
202/732-2399 
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Evaluating Sec, 353 Administration 

How a State administers its 353 program determines to a large degree 
whether its projects will be, or. even can be, evaluated. Field 
consultants have recommended the following techniques to help States 
assure project productivity and accountability: 

1. Review the State Plan to assure that 353 priorities will help 
achieve State goals, 

2. Consider the refunding of priorities not met by previous 
projects, 

3. Analyze the need for both Statewide and individual local program 
impact from 353 project results, 

4. Assure a realistic level of funding in relation to project 
objectives, 

5. Publish guidelines that... 

o Provide a clear statement of background ^ need, and expected 
outcomes for each priority to be funded 

o Distinguish between Statewide vs. local impact intents 

o Identify and give a rationale for project funding levels 

o Distinguish between compliance reports vs. products to be 
disseminated 

o Provide specifications for project evaluation 

o Clearly identify intended content of products, 

6. Require quarterly progress reports or formative evaluation-- from 
projects, 

7. Conduct site visits to projects at least twice yearly, 

8» Provide for the summative evaluation of projects. 

9. Provide for a third party (external) evaluation of projects that 
will be disseminated throughout the State. 

These, and other State administrative activities are included in *the 310 
Self Assessment guide (see /^pendix A) . 

A Self-Study Guide for Section 353 administration will be available in 
June, 1988. This guide should be useful to State staffs in evaluating 
and^ planning for the improvaiient of special project and teacher training 
efforts ♦ 
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Evaluating Special Projects and Training Programs 

An evaluation guide was recently developed under a Federal Vocational 
Education contract. It includes the following exhibit: 



STEPS IN THE EVALUATION PROCESS 



1. Define the 

Purpose and Scope 
of the Evaluation 



2. Specify the Evaluation 
Questions 



3, Develop the Evaluation Design and 
Data Collection Plan 



X 



A. Collect the Data 




5. Analyze the Data 

and Prepare the Report 



6. Use the Evaluation 
Report for Program 
Improvement 



(see Appendix B) 
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An Evaluation Guide for A,E, Programs / 
graphic to illustrate the relationship 
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from a 1984 310 project, offers this 
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EVALUATION 



In this context, the tern "evaluation" refers to the process of selecting, 
collecting, arxi interpreting infonwation needed for decision making. It is 
essentially research applied to decision making. As such, evaluation goes 
hand-in-hand with program development. "Program development" is defined as a 
systematic process for creating or modifying methods, procedures, and/or 
materials to be applied toward the achievenent of certain specified objectives. 

Most successful prograiiis follow the same cycle over the course of their 
developTient. Underlying the entire program developiisnt cycle is the evaluation 
process. Evaluation serves to provide objective information and fee3back to 
each step vithin the program developnent cycle. This use of evaluation would 
relate most directly to 363 "Demonstration Projects" (see Appendix A) . 

"Experimental Projects", of course, require a more extensive, more technical 
evaluation process. Appendix A also describes a Validation Guide for A.E. 
Projects that outlines processes, approaches, and criteria for project 
validation. 

Projects that are intended to have Statewide impact should involve an outside, 
"third party" evaluation in their plan. The National Adult Literacy Project 
suggested the following criteria for selecting outside consultants: 

(1) Evaluators are trained in and feel comfortable using several 
different evaluation methodologies, 

(2) Evaluators have a collaborative working style; they will work with 
programs to determine the array of measurable outcomes that are 
important in the view of the individual programs, 

(3) EvaluHtors are adaptable; they can adjust »±eir methods to the 
unique needs of an adult education environment. 
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(4) Evaluators are experienced in working in alternative education 
settings / preferably adult settings. 



Many special projects produce some sort of "product": a handbook, 
curriculum, guide, or other potentially useful material. Unfortunately, 
many of the publications are not well evaluated or field tested . Some 
are poorly packaged. The Focus project in Pennsylvania has reviewed 
dozens of 310 products from a variety of States. Focus rates these 
materials on: 

- Effectiveness - objectives and outcomes clearly stated, materials 
linked to results, content appropriate for target audience, 

- Innovation - addresses major priorities, creative use of resources, 
content not cOTinercialiy available, 

- Adoptability - clearly written, little staff training needed to 
transfer. 

A major new evaluation resource has just been published by a New York 310 
project. The Source Book for Evaluating Special Projects identifies 
five types of projects and related evaluation goals. 

1) Developmental Programs can be thought of as entrepreneurial 
activities. They are by nature risky, and they frequently aren't 
successful. However, they teach us a great deal about what does 
work, and eventually that knowledge can be used to develop a model 
that is successful. Evaluation in a Developnent Project focuses on 
gathering information on the possibilities of the program. 

2) The purpose for evaluating Demonstration Projects is to assess 
their performance in field conditions. We want to know if the 
program works well enough to implement Statewide. Vfe also want to 
know the performance standards for the program so that it can be 
monitored effectively when it is installed on a wider basis. 

3) For Installation Projects , evaluation and monitoring efforts are 
applied to determine how well these programs are meeti.ng the 
standards developed in the Demonstration phase.- When programs do 
not meet these standards, it is up to managers at the program site 
and the funding agency to do something about it. When managers are 
given valid and reliable tools to judge their organization's 
performance against that of some standard, they are eager to use 
then and incorporate then into their management procedures. 

4) In Training Projects , evaluators want to know if the training 
episode was effective in transferring the knowledge, skills and 
attitudes of the curriculum and if the trainer made the best 
possible presentation of the material. The evaluator may also want 
to know if the training has had an effect on performance in the 
work setting. In some cases, when the training will be repeated 
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over time as a standard item (e.g., training in prescrip- 
tive/diagnostic instruction) , the evaluation may help improve the 
curriculum. In these cases, assessment of the power of the 
training to affect program performance will become especially 
important. 

5) Planning and Evaluation Projects are usually episodic. The 

evaluation of these projects involves the validity and reliability 
of the process and whether the process produces the desired 
results (see Appendices A and C ) . 

The New York Source Book also offers advice on writing an evaluation 
report. 

The report should be written in clear, readable language. It must be 
accessible to an audience with varying degrees of knowledge about 
A.B.E. and evaluation methodology. The report will be used for 
different purposes, it should include a sum[nary or condensed version 
of the evaluation study for the casual reader, as well as a full 
presentation of relevant findings for the serious reader. 

A typical report includes the following sections: 

1. Program overview, including its history, rationale, goals, 
objectives, target groups, components, 

2. Description of the evaluation study, including the evaluation 
questions and the evaluation design, 

3. Results of findings, including presentation of data tables or 
illustrations, 

4. Discussion of the results in terms of the program's 
effectiveness and limitations, including discussion of any 
limitations to the interpretation of results, 

5. Conclusions and recomnnendations related to the programs, 

6. Copies of non-standardised instruments used in the evaluation 
in an appendix. 

Another res-)urce is available to evaluate, and plan for improvement of, 
staff training programs. 

The Self-Study Guide for ABE staff development programs was designed to 
help staff trainers and State staff development coordinators assess, plan 
and better understand their staff developiient programs. The guide 
contains six General Principles addressing collaborative planning, 
institutional policy, conditions of training, training processes, 
indiviual learner needs, and evaluation. 
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The Self-Study components include a set of 3-7 diagnostic questions 
relating to each principle, a section for planning the application of the 
principles, a section for assessing performance and a page for planning 
program improvenient activities (see Appendix D) . 

Evaluating the Impact of 353 Projects 

Since the responsibility for Adult Eauca*:ion Special Projects and Staff 
Training was transferred to States in 1974, few States have systemati- 
cally assessed the iirpact of their 310 programs. However, scxne States 
have conducted reviews of 310 projects, products, and the usefulness of 
projects to the field. 

Florida funded a study of the "Impact of 310 Projects, 1980-84 (see 
Appendix A) . 

The general purpose of the study was to determine the scope and impact of 
310 projects on adult education in Florida. Specifically, the study 
provided information on: 

1. How well 310 projects have helped to improve instructional and 
administrative skills of adult educators, 

2. How well 310 projects have helped to increase the knowledge of 
adult educators in methods, techniques and materioils available for 
adult education, 

3. The degree to which 310 project exenplary programs or models have 
helped to promote more effective adult educational practices, 

4. How well 310 project products and processes have been effectively 
COTtnunicated to the appropriate target population, 

5. The effectiveness of projects design and implanentation 
procedures, 

6. Possible areas of enphasis in which adult educational efforts hav^ 
not been effective, and 

7. The scope of 310 projects funded yearly- 
Three strategies were used to conduct the study: survey questionnaire, 
case studies and project reviews. The survey questionnaire was 
constructed from the research objectives and administered to adult 
education administrators and teachers. Nineteen projects were selected 
for case studies based on the criteria of "good examples" and "not so 
good examples" of 310 projects, type, size, cost and geographical 
location. Projects from 1980-1984 were reviewed to determine their scope 
and how well national and state priorities for adult education were 
addressed • 
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The study team developed seven recormendations to help improve future 
projects. These recommendations are good advice for all States 353 
programs: 



!• Assessment 

An assessment should be required for all 353 applicants to 
establish the problem/ examine constrains (human and material) , 
and identify i.tethods-means to implenents the project, 

2. Goals and Objectives 

Goals and objectives should be derived from needs assessment 
results. There should be a logical transition from goals or 
purpose to objectives, and from objectives to activities. The 
objectives should be specific enough to be measurable, 

3* Evaluation 

Evaluation should be conducted to assess the effectiveness of the 
projects, 

4. Follow-up 

Follow-up plans should be included to measure impact of the 
project, 

5. Disseminatior. 

A plan for dissoninaticn should go beyond just mailing out 
information to all local directors, to include identifying the 
primary audience and informing the audience about the nature of 
the products, 

6. Materials 

Materials which are produced should not only be of good quality 
but should be of the appropriate reading level and graphic layout 
to enable users to benefit from the materials. T^se materials 
should be usable by other adult education programs, 

7. Documentation 

It is important that project activities be documented 
systenatically. The record should include names and addresses of 
participants and other routine records of a project. Care should 
be taken to document the strengths, weaknesses, problens 
encountered, and to recommend how the process may be improved for 
future adaptation or adoption. 
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Evaluating impacts of staff training programs is often a difficult task. 
These documents desrribed in Appendix A offer examples and techniques for 
staff training evaluation and impact assessment: 

1 . Evaluation Guide for A.E. Programs / 

2c Source Book for Evaluating Special Projects . 

Conclusions 

The evaluation of Special Projects and Staff Training Programs should be 
considered an iniportant component of a State's 353 program* 
Accountability is seldom inexpensive/ and requires considerable 
imagination/ time/ and other uses of SEA and field resources. But the 
payoffS/ in terms of program improvement/ project information gathering/ 
field (and even national) credibility/ and increased or at least 
maintained funding/ will be well worth the effort. 
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Bibliography 

!• 310 Self Assessment for Adult Education Administrators is an 

instrument to assist state Mult Education staff in planning and 
improving their administrative systems. It includes 57 questions in 
five administrative areas: Setting Priorities, i^lication Process, 
Guidelines, Project Monitoring, and Dissemination. Developed by the 
Division of Adult Bd'^cation, U.S. Dept. of Education, 1984, 22 
pages. 

2. Evaluation Guide for Adult Education Programs includes strategies and 
techniques for A.E. program evaluation. It addresses The Stages of 
Evaluation Process, Evaluation resign. Methods of Data Collection, 
Statistical Analysis Procedures,, and Reporting E^/aluation Results. 
Developed by Research for Better Schools, Philadelphia, PA under a 
New Jersey 310 grant, 1934, 75 pages. 

3. Validation Guide for Adult Education projects covers a Rationale for 
Validation, Characteristics of Validation Models, and Issu:>s xu 
Validation such as adoption vs. adaption, educational vs. statisti- 
cal significance, program staff responsibilities, and costs of vali- 
dat.ion. Developed by Research for Better Schools under a New Jersey 
310 grant, 1984, 23 pages. 

4. S ource Book for Evaluating Special Projects includes a presentation 
of evaluation approaches for five types of 353 (310) projects: 
Development, Demonstration, Installation, Training, and Support 
Projects. It also contains techniques and instruments for evaluating 
various projectr>. Developed by Intarorganizational kelationships, 
LTD., Altamont, NY under a New York 310 g.:ant, 1988, 414 pages plus 
Appendices. 

5* Impact of 310 Projects in Florida, 1980-1934 :x)vers The Methodology 
of the scope - of - impact study. Strengths and Weaknesses of 310 
projects, and other findings of surveys, case studies, and project 
reviews. Recommendations for improvement of Florida's 310 effort are 
offered. Study conducted by Florida A & M University under a 310 
grant, 1985, 27 pages (abridged) . 

6« Effective ABE Staff Development - a Self-Study Guide presents 28 
diagnostic questions illustrating six General Principles: Planning 
Involvenent, Support Policies, Training Conditions, Mastery, 
Individual's Needs, and Evaluation. The guide includes worksheets 
for diagnosis, planning of activities, and program assessment and 
improvement for each principle. Developed by Thv. Division of Adult 
Education, U.S. Dept. of Education, 1987, 19 pages. 
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/APPENDIX B 



EVALUATION PROCESS AND PLANS 



Evaluatioa Is one of the basic tools for program management and improvement. The 
central questions to be answered include: 

• To what extent have program objectives been met? 

• ^That contributed to the successes and failures? 

• What changes and improvements should be made? 

Thus, evaluation provides inforaation for documenting and improving program 
effectiveness. 

The documentation of program effectiveness involves the systematic collection, 
analysis, and reporting of information or data. It should not be forgotten, 
however, that the improvement of program effectiveness involves UuRaa judgement, 
i.e., using the reported data for planning and decision-making ;hjxa will result in 
program improvement and increased effectiveness. 

A. Overvievr of the Evaluation Process 

The evaluation process can be described as involving six progressive steps. 
These steps are shown in Exhibit 1, and are discussed below. It is important 
to remember that initiating an evaluation cannot wait until a program is 
nearing completion. An evaluation should be planned as part of the overall 
program and should be implemented at the same time the program begins 
operation. In this manner, program activities can be adequately documented 
from their beginning, and baseline data on program participants can be 
collected as they enter the program. 

Step 1 ; Defining the Purpose and Scope of the Evaluation 

The first step in planning an evaluation is to define it? purpose and 
scope. This helps set the limits of the evaluation, confining it to a 
manageable size. Defining its purpose includes deciding on the goals 
and objectives for the evaluation, and on the audience who will use 
the evaluation results. The evaluation goals and objectives may vary 
depending on whether the program being evaluated is new and is going 
through a try-out period, or if a program has been thoroughly tested 
and needs documentation of its success before information about the 
program is widely disseminated and adoption by others encouraged. 
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EXHIBIT 1 
STEPS IN THE EVALUATION PROCESS 



1« Define the 

Purpose and Scope 
of the Evaluation 



2. Specify the Evaluation 
Questions 



3. Develop the Evaluation Design and 
Data Collection Plan 



A, Collect the Data 



5. Analyze the Data 

and Prepare the Report 



6. Use the Evaluation 
Report for Program 
Improvement 
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Depending on the purpose, the audience for evaluation may be 
restricted to the project director and his/her staff, or may include a 
wider range of individuals, from agency administrators and 
decision-makers, to planners and other officials at the local, state, 
or federal level. 

The scope of the evaluation depends on the evaluation's purpose and 
the information needs of its intended audience. These needs determine 
the specific components of a program which should be evaluated and on 
the specific project objectives which are to be addressed. If an 
evaluation of £ program has recently been conducted, a limited 
evaluation may be designed to target on certain parts of the program 
which have been changed, revised, or codified. Similarly, the 
evaluation nay be designed to focus on certain participant objectives 
which were shown to be only partially achieved in a previous 
evaluation. Thus, this step will define exactly which aspects of the 
program are to be evaluated. Costs and resources available to conduct 
the evaluation must also be considered in this decision. 

Step 2 ; Specifying the Evaluation Questions 

Evaluation questions are general questions that grow out of the 
purpose and scope specified in the previous step. They help further 
define the limits of Che evaluation. The evaluation questions are to 
be discussed and answered in the evaluation report and should be 
formulated to address the needs of the specific audience to whom the 
evaluation is directed. Evaluation questions should be developed for 
each component of the program which falls into the scope which was 
defined in tiie previous step. For example, questions may be 
formulated which concern the adequacy of the curriculum and the 
experience of the teaching staff; other questions may concern the 
appropriateness of the vocational skills being taught in relation to 
employment opportunities in the local community; additional questions 
may relate to the appropriateness of the individuals being recruited 
for the program with respect to their English language proficiency and 
employment str-'us; and finally, evaluation questions may relate to the 
extent to which participants are achieving the goals of the program, 
such as stable employment and Increased earning capability. 

A good way to begin formulating evaluation questions is to carefully 
examine the project *s objectives; another source of questions is to 
anticipate problem areas concerning program implementation. 
Importantly, the audience for or expected users of the evaluation 
should be involved in developing the evaluation questions. This 
should never be left solely to the outside evaluator, no matter how 
familiar he or she is with the program. Once the evaluation questions 
are developed, they should be prioriti.ed and examined in relation to 
the time and resources available. Once this is accomplished, the 
final set of evaluation questions can be selected. 



ERIC 



DETSLOPMXNT ASSOCIATES. INC. 



4 



Stop 3! Developing the Evaluation Design and Data Collection Plan 

This step involves specifying the approach to answering the evaluation 
questions, including how the required data will be collected. This 
will involve: 

- specifying the data sources for each evaluation question; 

- specifying the types of d^ta collection approaches snd instruments 
needed; 

- specifying the specific time periods for collecting the data; 

- specifying the staff riembers who will be assigned to collecting the 
data, and how their data collection responsibilities relate to 
their other project responsibilities; and 

- specifying the resources which will be required to carry out the 
evaluation. 

The design and data collection plan is actually a roadmap for carrying 
out the evaluation. An important part of the design is the 
development or selectio.i of the instruments for collecting and 
recording the data needed to answer the evaluation questions. Data 
collection Ins tuments may vary from record-keeping forms, 
questionnaires, interview guides, to vocational and language skills 
tests. Some of the instrument ition may already be available. I.e., 
forms used for recordkeeping and management purposes, such as 
recruitment or intake forms, etc. Some of these forms will have to be 
modified to meet the evaluation needs. In other cases, new 
instruments will have to be created. 

In designing the instruments, the relevance of the items to the 
evaluation questions and the ease or difficulty of obtaining the 
desired data should be considered. Thus, the instruments should be 
reviewed by the project director and staff member? to ensure that the 
data can be obtained in a cost-effective manner, and without causing 
major disruptions or inconveniences to the project. 

Step 4 ; Collecting the Data 

Data collection should follow the plans developed in the previous 
step. The individuals assigned to the various data collection tasks 
need to be thoroughly trained in the data collection requirements and 
procedures. Only by following standardized procedures will the data 
be reliable and valid. Following training, the project director needs 
to monitor the staff to ensure that they are accomplish.ing their data 
collection assignments according to the specified time schedule. The 
data should be recorded carefully and neatly so they can be read and 
interpreted during the analysis stage. Proper record-keeping and 
filing are similarly important so that the data are not lost or 
misplaced. Any problem should be discussed with the project direc r 
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and evaluator. Deviations from the data collection plan should be 
docuiaentefl 20 that they can be considered in analycirkg and 
interpreting the data. 

Step 5 ; Analyzing the Data and Preparing a Report 

This sfep involves tabulating, suomariziag, and interpreting the 
collected data in such a way as to answer the evaluation questions. 
These procedures should be compatable with the type and amount of data 
which were collected, and the goals and objectives of the evaluation. 
Appropriate descriptive measures (frequcacy and percentage 
distributions, central tendency and variability, correlation, etc.) 
and inferential techniques (significance of difference between aeans 
and other statistics, analysis of variance, ch~square, etcw should be 
used to analyze the data. The local evaluator should have 
responsibility for t^is aspect of tho evaluation. 

The evaluation will not be completed until a report has been written 
and the results communicated to the project director and other 
appropriate administrc't-ors and decision-makers. In preparing the 
report, the nriters should be clear about the audience for whom the 
report is b^xng prepared. Two broad questions need to be considered: 
(1) Wha.t does the the audience need to know about the evaluation 
results? and (2) How can these results be best presented? Different 
audiences need different levels of information. Administrators need 
general information for policy decision-making, while project staff 
may need more detailed information vrhich focuses on project activities 
and effects on participants. 

The report should cover the following;: 

" The goals of the evaluation; 

- The procedures or methods used; 

- The findings ; and 

- The implication of the findings inclualng recommendations for 
changes or improvements in the program. 

Importantly, the report should be organized so that it addresses all 
of the evaluation questions specified in Step 2* 

Step ? ; Using the Evaluation Report for Program Improvement 

The evaluation should not be considered successful until its results 
are used by program managers and decision-makers for program 
improvement. After all, this is the ultimate reason for conducting 
the evaluation. The evaluation may indicate that a program activity 
is not being implemented according to plan, or it may indicate that a 
particular objective is not being met. If this does occur, it is then 
the responsibility of the project director to make appropriate changes 
to remedy the situation. Project directors should never be satisfied 
with their programs. Improvements can always be made, and evaluation 
is an important tool for accomplishing this purpose. 



17 



ERIC 



MMlrfffTmiH ■ DBVXLOPMJeNT ASSOCUTES* INC. 



B, Planning the Evaluation 

The evaluation should be conducted by an Independent, experienced evaluator. 
This Individual will provide the expertise for an evaluation which is 
comprehensive, objective, and technically sound. The project director and 
her/his staff must work closely with the evaluator beginning with the planning 
stage to ensure the evaluation meets the exact needs of the project* 

Adequate time and thought for planning an evaluation Is essential, and will 
give the project director and staff an opportunity to develop ideas about what 
they would like the evaluation to accomplish. The evaluation should address 
the goals specified in the project application and management plan. In some 
projects, however, one or more goals or objectives may require special 
attention. Some activities or instructional strategies may have been recently 
Implemented, or the staff may be aware of some special problems which should be 
addressed. For example, there might have been a recent breakdown in 
consmunlcauion between the ESL teachers and the vocational instructional staff; 
or the characteristics of the participants in recent training cycles might have 
begun to differ significantly from past groups, having implications for 
vocational training or the approach to language instruction. These are 
examples of things which should be considered when the project director selects 
an evaluator. The evaluator must then familiarize himself or herself with the 
special issues of concern on which the evaluation should focus. 

Thus, the initial step of the evaluation process involves thinking about any 
special needs which will help in planning the overall evaluation and selecting 
the evaluator. Special evaluation questions and problems identified in the 
instructional staff area might r Jggest that an evaluator is needed who has 
expertise in evaluating Instructional systems, etc. Similarly, if the project 
needs an evaluation which requires frequent on-site observation of teaching 
methodologies by the evaluator, then this will help the project director focus 
on hiring someone located nearby so that travel costs can be kept to a minimum. 
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la summary, defining the scope involves setting limits, identifying specific 
areas of inquiry and deciding on what parts of the program and on which 
objectives the evaluation will focus* The scope does not answer the question 
of how the evaluation vill be conducted* In establishing the scope, one is 
actually determining which components or parts of the program will be 
evaluated. This step is important and, indeed, implies that the evaluation may 
not cover every aspect and activity of the program* 



C. Selecting the Svaluator 



Selecting an evaluator for the program is one of the most important elements in 
ensuring a technically sound and useful evaluation. The basic criteria 
suggested for selection are as follows: 



• Skills in evaluation design, constructing data collection instruments, 
collecting data, managing and maintaining quality control over data 
collection, analyzing data, and writing reports; 

• Experience in conducting evaluations of vocational training projects 
targeted at LEP adults; 

• Knowledge of the BVT model; 

• Experience in collecting data from employers and community and business 
groups ; 

• Ability and willingness to work directly with the project director in 
order to design the evaluation, oversee its implementation, and prepare a 
report ; 

• Reside within reasonable distance of the project so that travel costs are 
minimal and scheduling work sessions is ^ot a problem; and 

• Available for complete time period required for the project at a rate 
that fits the basic budget resources allocated to evaluation and the 
number of evaluator-days projected in the initial planning; the project 
might expect to plan for at least 20-25 work-days for the evaluator* 

Once an individual has been selected and has agreed to become the local 
evaluator for the project, a contract and work plan murt be developed so that 
expectations, roles, and responsibilities are clear to all parties. 
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Specifying the Roles of Project Director, Staff and Svaluator 



In order for the evaluation to be planned and carried out effecf^vely , the 
roles of the project director, st.'.rf , and evaluator must be made clear to all 
parties. The evaluator should be responsible for specifying the design and 
developing the data collection instruments, training project staff to collect 
the required data, analyzing the data, and preparing a comprehensive report. 
The project director should work with the evaluator in specifying the 
objectives and scope of the evaluation, and assigning and supervising the 
project staff in carrying out their assigned data collection and record-keeping 
tasks. The project staff should be assigned the major data collection and 
record-keeping tasks for the evaluation. This will increase their job 
responsibilities, buf is the most cost-effective way of collecting the required 
data. More specifically, the suggested roles of the project director, the 
evaluator, and the project staff are listed below: 

The project director will: 

t recruit and hire an experienced evaluator; 

• work with the evaluator to define the objectives and scope of the 
evaluation; 

• work with the evaluator to define Ju^ evaluation questions which will be 
addressed ; 

• review and approve data collection instruments and procedures to ensure 
that the}} are compatible with project activities; 

• assign and supervise project staff for data collection and recordkeeping; 

• approve schedule and expenditures for the evaluation and 

• approve final evaluation report. 
The evaluator will? 

• work with the project director to define the purpose and scope of the 
evaluation; 

• work with the project director ';o specify the evaluation questions which 
will be addressed; 
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• develop the evaluation design; 

• develop data collection instruments and procedures, including the 
schedule and calendar for all data collection; 

• train projact staff to collect the required data; 

• analyze the collected data to answer the evaluation questions; and 

• prepare the evaluation report. 
The project staff will: 

• collect the evaluation data; 

• maintain participant tracking records as well as conduct follow-up 
surveys of former trainees; and 

• assist the evaluator in tabulating and summarising the collected data. 
£• Allocating Resources 

Critical questions in planning the evaluation concern how much staff time an*"' 
financial resources can be expended on the evaluation, and what resources are 
needed to actually carry it out. These two questions need to be addressed by 
the project d-'rector and the evaluator early in the design process. Many 
evaluators will raise these questions In the initial explorat;^ry discussions 
concerning the evaluation. Preliminary planning will have to be accomplished 
before a final decision can be made on both the role and level of effort of the 
evaluator and each project staff member. 

A variety of design decisions must bie M-ie during the planning stages that 
affect the allocation of both financial and staff resources. Each decision 
affects the staff /evaluator rules and functions, level of effort, and 
resources, and ultimately determines the overall scope of the evaluation. For 
instance, a record-keeping procedure must be developed to maintain accurate 
evaluation data. A tracking system Is also needed to manage follow-up surveys 
of former trainees. These record-keeping and tracking systems, as well as the 
actual data collection activit:ies, involve starf and material costs. These 
costs need to be considered in defining the final scope of the evaluation. 
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The purpose of Adult Basic Education Special Projects funded under Section 310 
of the Adult Education Act, P.L 91-230 and Amendments, is to strengthen adult 
literacy services through experimentation with and dissemination of innovative in- 
structional methods, programs, and opera^'mal and administrative systems. 

Based on a review of proposals and summaries of Section 310 projects funded 
over the past several years, we have developed a typology of 310 projects using five 
major categories. These categories have implications for evaluation and monitoring 
approaches. The first three categories correspond to three stages in program design: 
Development, Dcmonslrntlon and ImplemcntRtlon.The last two categories. Training 
and Planning and Evaluation, provide support functions for the actual programs. 
The five-category project typology is shown below: 

1. Dcvclonment - Projects funded in the Program Development category in- 
volve planning and/or pilot testing new and innovative approaches to adult 
education. 

Innovation occurs when someone decides to try something different. The in- 
novation can be an application of an idea or an approach from another 
program, a variation on the current program, or a new approach based on the 
latest research. The most appropriate sites for Development Projects are 
agencies with creative staff who already are developing innovative programs, 
often with an evaluation component. Examples of potential Development 
Projects might include a mini-grant for the use of video in ESL classes or a 
vocational program for GED students. Approaches to evaluating Develop- 
ment Projects are discussed in Chapter 3. 

2. Demonstration - Projects in the Demonstration category usually involve 
implementation and further refinement of innovative program approaches 
that have shown promise of success through initial pilot testing. 

Demonstration Projects generally implement among broader population 
groups and geographic areas those model programs initially developed ^ith a 
more limited number of people. These projects are of greatest value to the 
overall service system when they include a formal evaluation of their effec- 
tiveness and efficiency. Demonstration Projects can also assess the wisdom of 
replicating program models on a statewide basis and establish criteria for 
monitoring installation of successful program models. Sites most suitable for 



Demonstration Projects include large agencies serving heterogeneous 
populations and those v.'ho have well-developed evaluation resources. 
GRASP and Job Clubs are good examples of projects in this category. Basic 
guidelines for evaluating these programs are found in An Administrator's 
Evaluation Guide for Attaining Validation in Adult Education, available 
from the New York State Education Department, Bureau of Adult and Con- 
tinuing Education. 



3. Instullation - Projects in the Installation category facilitate expanded use of 
program models whose performance has been validated in Demonstration 
Projects. 

Installation Projects provide implementation of successful program models 
within pre-established guidelines, and these projects are monitored to insure 
conformity to program replication criteria (e.g. use of multi-disciplinary cur- 
riculum). These programs benofit the overall service system by widening the 
application of innovative approaches to adult learning. Sites most appropriate 
for Installation Projects arc agencies serving high-need areas. Section 310 
monies would probably be used most appropriately to fund only the start-up 
of projects. The evaluation of these projects is discussed in The Datn iase 
Primer, available from the New York Slate Education Department, Bureau of 
Adult and Continuing Education. 

4. Training - Training Projects pr(^v^dc new knowledge and sicills for service 
providers and program managers. 

An essential role of Training Programs is to provide the knowledge and skills 
necessary to facilitate the shift of successful program models from develop- 
ment to demonstration and from demonstration to installation and to help 
staff maintain the knowledge and skills required for continued successful 
program performance. Examples of funded projects in this category are 
statewide training projects in marketing to the hard-to-reach and the regional 
staff development projects. The evalu'Jtion of these projects is handled in 
Chapter 4. 

5. Evaluation. P lannin g nnd other Sunnort Projects - Projects in this category 
are specific activities designed to improve the management practice. 

Funded projects in this category might include strategic planning and evalua- 
tion of the impact of CAI on outcome measures. Tlie evaluation of these 
projects is addressed in Chapter 5. 

A simplified diagram of the.projcct typology is shown in Figure I. 
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FIGURE 1. PROJECT TYPOLOGY 



TRAINING 






TRAINING 




T 


D 


T 


I 


D 


R 


E 


R 


N 


E 


A 


M 


A 
/\ 


S 


V 


I 


0 


I 


T 


E 


N 


N 




A 


L 


I 


S 


I 


L 


0 


N 


T 


N 


L 


P 


G 


R 


G 


A 


M 




A 




T 


E 




T 




I 


N 




I 




0 


T 




0 




N 






N 






EVALUATION, PLANNING, AND OTHER SUPPORT 



This framework helps us to sort ova tlic kind of evaluation and monitoring ac- 
tivities to use for the various types of special projects. Even though there will |)robab- 
ly always be special projects that do not fit neatly into these five categories, most 
projects will incorporate parts of one or more types. By examining the areas of over- 
lap with these categories, those designing evaluation and monitoring approaches for 
projects that do not conform to the proposed categories should be able to find useful 
and relevant information and examples. 
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THE PURPOSE OF THE EVALUATION PROCESS 



la order for evaluation efforts to be successful in a complex organizational en- 
vironment, it is essential that central participants have a shared concept of the pur- 
pose for undertaking evaluation activities. A primary' goal for evaluation efforts, one 
to which wc believe all pa ticipants in the process can subscribe, is ''to improve the 
effectiveness of the service delivery system." 

Developmental Programs can be thought of as cntrenrencuriu! activities. They are 
by nature risky, and they fre(iucntly aren't successful. However, they teach us a great 
deal about what docs work, and eventually that knowledge can be used to develop a 
Jiiodcl that is successful. Evaluation in a Development Project focuses on gathering 
information on the possibilities of the program. 

The purpose for cvaluuling Demonstration Projects is to assess their performance 
in field conditions. Wc wani to know if the program works well enough to implement 
.statcvjidc. We also want to knov/ the performance standards for the program so that 
it can be monit^jrcd effectively wlicn it is installed on a wider basis. 

For Installation Projects, evaluation and monitoring efforts arc applied lo deter- 
mine how well these programs arc meeting the standards developed in the 
Demonstration phase. When programs do not meet these standards, it is up to 
nianagcrs at the program site and the funding agency to do something about it. When 
nianagcrs arc given valid and reliable tools to judge their organization's performance 
against that of some standard, they arc eager to use them and incorporate them into 
their management procedures. 

With few exceptions, the Training, ICvaluntlon and Planning activities arc not 
continuous. In Training Projects, cvaluators want to know if the training episode was 
effective in transferring the knowlcgc, skills and attitudes of the curriculum and if the 
trainer made the best possible prcscniation of d)e material. The evaluator may also 
want to know if the training has had an effect on performance in the work setting. In 
some cases, when the training will be repeated over time as a standard item (e.g., 
training in prescriptive/diagnostic instruction), the evaluation may help improve the 
curriculum. In these cases, assessment of the power of the training to affect program 
performance will become especially important. 

Planning and Evaluallon Projects are usually episodic. The evaluation of these 
projects involves the validity and reliability of the process and whether the process 
produces the desired results. 



We hope the above discussion has helped you see how a systematic and coor- 
dinated approach to the evaluation of Section 310 projects will benefit everyone. 
There will be substantial gains in system accountability at both the state and local 
levels. The data obtained through this evaluation and monitoring process would 
allow the Bureau of Adult and Continuing Education to document why monies were 
spent on a particular program mix and what outcomes were achieved. Data obtained 
from the more formal evaluation process in the Demonstration phase will prevent in- 
stallation of programs that are not worth the investment and could be used to project 
the statewide impact when these projects move into the Installation phase. At the 
local level, program administrators will have the data to justify decisions about 
cniploying one program approach over another and to draw conclusions about out- 
comes. Teachers and counselors can monitor their programs, and students can be as- 
sured they are using a valuable service. 



EVALUATION APPROACHES 

In this section wc will discuss the most appropriate evaluation approaches and iic- 
livilies for cadi project category. An overview of suggested evaluation approaches 
for Development Demonstration, and Installation Projects is shown in Table 1 
b^.ginning on page 8. While a detailed explanation of all these evaluation methods 
and techniques is outside of the scope of this source book, wc wil! discuss our ration- 
ale for selecting the Jj^proachcs for each project typology. Several of these ap- 
proaches will be developed in the subsc(|ucnt chapters. 

Evaluation, especially fornial evaluation, is a costly undertaking. The resource^ for 
evaluation are liniitcd, and they must be used in an efficient way. Because die 310 
Projects have a focus on innovation, they can justify spending a higher proportion of 
resources on evaluation than programs designed to operate with established inter- 
vention strategies. Nonetheless, even in the Section 310 program, the percentage of 
resources expended on evaluation should be small compared to the expenditures on 
student servi::es. 

The most effective method for distributing the resources available for evaluation 
within the A.B.E. system is to concentrate the most sophisticated and methodologi- 
cally advanced efforts on Demonstration Projects. It is at the conclusion of t!ie 
Demonstration phase that the funder must decide and justify whether a program 
model should be considered for statewide replication. Here is where evaluations, 
even fairly expensive ones^ can have the greatest payoff. Since the outcome of the 
decision to implement a program model statewide has mjjor funding implications, 
this decision should be uv^Oz with the support of the best and most persuasive infor- 
mation obtainable. 



Development Programs, on the other hand, are less likely to benefit from ^orm^\ 
evaluation procedures. These programs often have small budgets, and they do not 
serve a sufficiently large number of students to warrant sophisticated quantitative 
evaluation approaches. The decision about funding a Demonstration phase for a 
model that has been through e Development phase typically can be justified ade- 
quately with impressionistic rather than objective data. 

Due to the large expenditure of resources for Installation Programs, there are 
legitimate demands to prove tliat these programs are achieving the results intended 
at a reasonable cost. However, if rigorous procedures have shown the effectiveness of 
a program at the Demonstration phase, that knowledge can be employed in designing 
much less expensive yet still valid strategies for monitoring a program's performance 
throughout its Installation and operating cycle. 

Most Training, Planning and Evaluation Projects are limited in purpose. With 
few exceptions, the Training Projects arc time-limited and repeated infrequently. 
Typically, some resources arc reserved to evaluate the adequacy of the process. 
Then, if the desired improvement in the system docs not occur, the process itself xan 
be eliminated as a reason for failure. 

With these points in mind, we will elaborate briefly on the evaluation and 
monitoring activities for the project types listed in Table 1. 

DEVELOPMENT PROJECTS 

A written project description that documents what occurred during the course of a 
project's operating period is the minimum step necessary to evaluate Development 
Projects. Descriptive material might include program manuals, curricula, reviews of 
research, instructions for replication, or staff training materials. This project 
documentation and other relevant information could be submitted for expert review 
and testimony to evaluate the efficacy of the actual or anticipated program ap- 
proaches or outcomes. 

An important step for every project in the Development group being considered 
for possible replication as a Demonstration Project is an evaluability assessment. 
This assessment would critically examine whether the proposed program model 
meets criteria required to conduct a more formal evaluation. When it has been deter- 
mined that a formal program evaluation is feasible, plans should be made to imple- 
ment evaluation studies by selecting and pilot testing the instruments and procedures 
that maybe used. 
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DEMONSTRATION PROJECTS 



For Demonstration Projects, evaluation activities should be considered an essen- 
tial component of the project and included as part of the project proposal and 
budget. Ideally, there would be a start-up phase to Demonstration Projects that 
would involve the finalization of evaluation designs and staff training for evaluation 
tasks prior to thc^ actual delivery of student services. The evaluation designs for 
Demonstration Projects should be comprehensive and sophisticated. They require 
the cooperative effort of project staff and Bureau staff and the possible use of outside 
consultants. Since Demonstration Projects usually involve several sites, the Bureau 
would serve in a coordinating role, assuring the comparability of methods used across 
sites. In many instances, it would be valuable to have staff with evaluation expertise 
available to work regularly with project staff at each site. 

An important aspect of the evaluation process for Demonstration Projects invol- 
ves examining the results obtained across all sites to assess the performance of the 
overall program model. Once the findings across program sites have been syn- 
thesized for specific program models, the summarized results sh nild be compared to 
competing program mi^dcls in order to choose the best alternative. This activity has 
become a subspcciality within the program evaluation field and is identified as meta- 
analysis. 

INSTALLATION PROJECTS 

When the decision has been made to institutionalize a program model through the 
funding of Installation Projects, th'^ level of effort expended on evaluation can be 
reduced sub.siantially. It is not necessary, for example, to reestablish at each installa- 
tion site that the intervention methods employed in the model program strategy have 
an impact on the desired long-range outcomes. Rather, evaluation of Installation 
Programs should focus on whether the effective program processes from the 
Demonstration phase are being successfully implemented. 

The evaluation experience gained in the Demonstration phase will be invaluable 
in designing the evaluaticm and monitoring systems for Installation Programs. The 
key data elements can be chosen and data collection procedures can be streamlined 
and implemented on a more limite. basis by using samples and posttest-only designs. 
These simplified requirements will be relatively easy for local agencies to implement 
by providing minimal training for staff with limited evaluation expertise. The data 
obtained in the Demonstration phase can also be used to develop standaros which 
will provide normative criteria forjudging how well sites are performing in installing 
a given progrii n model. 
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TABLE 1. EVALUATION AND MONITORING ACTIVITIES PROPOSED FOR 3 10 

PROCESS 



PHASE 


f 

ACTIVITIES 


Development 


Prepare descriptive documentation on innovative interven- 
tion approaches, including program manuals, instructions, 
curricula, and/or training materials (project sites) 

Review of applied research into causes of targeted 
problems or intervention approaches and their relation lo 
umdcd 310 projects (Bureau staff, consultants) 

Solicit expert testimony on the anti^ ipatcd or actual out- 
comes of innovative program strategies (Bureau or program 
sites, academics) 

Provide policy analysis relating 310 strategies to other in- 
itiatives within SED and other state agencies (Bureau and 
SI2D staff and/or state-level interagency groups, consult- 
ants) 

Conduct descriptive, informal case studies relating student 
outcomes to intervention processes; 100% samples of small 
student groups (project sites) 

Conduct evaluabllity assessments of program models being 
considered for demonstration: for example, are program 
outcomes well-defined and mc^isurabler (project sites with 
technical assistance from Bureau) 

Develop more rigorous evaluation approaches by designing 
and pilot testing procedures and instruments (project sites 
with technical assistance from Bureau) 
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TABLE 1. EVALUATION & MONITORING ACTIVITIES (CONTINUED) 



PHASE 


ACTIVITIES 


Demonstration 


Refine and expand evaluation procedures from the 
DevelopmentPhasc rior to Implementation of programs at 
demonstration sites (Bureau staff with possible addition of 
outside consultants) 

Employ forma! evaluation designs incorporating some or 
all of tlic following methods: • 

• controlled studies using random assignment or 
quasi-experimental designs 

• large samples with heterogeneous social groups 

#pre-post outcome measures, using some combina- 
tion of comprehensive testing, detailed interviews, 
staff or student questionnaires, etc. 

• client salisraclion studies and other community- 
based measures to assess program acceptance 
among the public 

• cost-accounting procedures to accurately docu- 
ment the program's use of resources 

• assessment of unintended consequences of 
program interventions through comprehensive ap- 
proaches to monitoring program Impact- 

• use follow-up investigations to measure ultimate 
program Impacts 

• studies coordinated by Bureau and carried out in 
conjunction with project staff and possibly outside 
consultants 

« 

Analyze data across demonstration sites to determine effec- 
tiveness of program model and to identify critical factors af- 
fecting program performance; disseminate findings through 
evaluation reports, newsletters, etc. (Bureau) 
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TABLE 1. EVALUATION & MONITORING ACTIVITIES (CONTINUED) 



PHASE 


ACTIVITIES 


Insiallntion 


Use meta-analysis lechniaues to synthesize findings across 
competing program models and to select most effective and 
efficient program models for installation (Bureau) 

Impiement^'a Umltcd set of prevlouslv developed procedures 
to evaluate the conformance of installation sites to tiie 
demonstrated program model (Bureau-designed, with 
project sites-supplying data) 

Develop a monitoring isystem at installation sites which 
would have some or all of the following characteristics: 

• use brief questionnaires, automated tests or other 
low-cost methods to obtain student outcome 
measures 

• collect data on a pos(test-only basis and for small 
samples to further reduce cost 

• install mnnngement inrormation systems that rely 
on aggregated data reported by project sites to 
document comprehensive program information 
such as number of students served, student charac- 
teristics, or number of services provided, results for 
student groups relative to standards 

• incorporate site visits by regional staff as a means 
of gathering additional monitoring information, to 
conduct program audits and to aid in the use of 
this information for program management at the 
local level 

Provide a system to process data centrally and distribute it 
to program directors and Bureau field staff on a timely 
basis. Reports would include standardized criteria (initially 
obtained in the Demonstration phase, subscaucntly as- 
sembled from Installation sites) against whicn to assess 
project performance 
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Apperdix D 

Principles and Diagnostic Questions frcm ABE Staff Development Self-Study 
Guide 

General Principle A: All segments of those involved in the ABE program 

are involved in planning for staff development. 

Diagnostic questions: 

1. Does your staff development program assess the perceived training 
needs of teachers, counselors, other staff? 

2. Is the local ABE program director involved in inservice programs? 

3. Have participants been involved in diagnosing, implementing and 
evaluating their own learning experiences? 

4. Have ABE programs in your area collaborated in the planning and 
implementation of staff development? 

General Principle B: Institutional policies support effective ABE staff 

development activities. 

Diagnostic questions: 

1. Is staff development an on-going process that encourages growth 
of continuing staff? 

2. Does staff development encourage new staff to beccxne an integral 
part of the ABE system and program development? 

3. Is the staff development program closely related to State or 
local priorities? 

4. Does your ABE program link training activities to a professional 
developnoent plan (ECP) or a general program effort? 

General Principle C: Training conditions facilitate effective staff 

development activities. 

Diagnostic questions: 

1. Are ABE toachers permitted to participate in staff development 
activities on "prime time" as part of their professional • 
assigrHT^ent? 

2. Are training sessions held in comfortable surroundings away from 
class sites? 

3. Does staff development provide positive incentives to 
participate, both during the activity and during its 



4. Have opportunities been provided during training sessions for 
small--group discussions of the application of new practices and 
sharing of ideas about effective instruction? 

5. Does the staff development allow teachers to experiment with new 
ideas when working in teams or with other group support? 

6. Does your staff development program enploy persons perceived as 
credible by the group in training? 

7. Are teachers who have been trained as staff developers involved 
in staff training activities? 

General Principle D: Staff development provides sufficient time and 

opportunity for participants to learn, practice, 
master and apply the training content. 

Diagnostic questions: 

1. Does your ABE staff development allow for intervals in which to 
plan and try out new approaches and return to evaluate their 
successes and problems? 

2* Has time for experimentation been provided while teachers adapt 
new practices to meet the needs of their students? 

3. Is personal, in-classroom assistance from administrators and 
fellow teachers available while proctices are being implemented? 

4. Are teachers given the opportunity to periodically meet and share 
ideas with colleagues? 

5. Have inservice programs been planned to accomodate both short 
and long range staff needs? 

6. Is there a plan or set of instructional materials ready for use 
when participants leave training activities? 

General Principle E: The staff development program recognizes individual 

learner needs and provides experiences to address 
these needs. 

Diagnostic questions: 

1. Do training activities permit variation in the ways the Tearners 
participate, and in ways they use what they learn? 

2. Does your staff development program provide different educational 
experiences for participants at different stages of develv^pnrtent? 

3. Are participants' concerns listened to, and appropriate pro" ram 
adjustments made? 
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General Principle F: Evaluation is an integral component of ABE staff 

development. 

Diagnostic questions: 

1. Does your staff development process provide continuous feedback 
on the effectiveness of training? 

2. Has the process employed a variety of evaluation techniques? 

3. Do evaluation results influence future planning and 
implementation of staff development activities? 
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